1. Field of the Invention
The present invention relates to an image processing technique, and more particularly to a technique for resolution conversion such as increasing the resolution of video data.
2. Description of the Background Art
With the spread of digital image devices such as digital video cameras and digital still cameras, it has become common that high-resolution digital images are handled by various types of input/output devices. Particularly, with still images, the image pick-up device of an ordinary consumer digital still camera has five megapixels or more, and some products include an image pick-up device having over ten megapixels. It can be said that sufficiently high resolutions have been realized for digital photography applications.
As the demands for higher resolutions for still images have been quite satisfied, it is now expected in the field of digital image applications that there will be more demands for higher resolutions for video data, particularly, for video data such as movies where each frame image is of such a high resolution that it can be used as a good-quality still image. There are two fields of application to which image resolution increasing techniques are applied, i.e., image input systems such as cameras and image display systems such as TV sets. The present invention is directed primarily to image input systems.
Increasing the resolution of video data in an image input system involves the following problem. Where video data of an HD (High Definition) TV or better quality is desired, it is very difficult to read out all pixels at a video data frame rate of about 30 fps due to the large number of pixels even if the image pick-up device has enough pixels. If such a high-speed read-out process is performed forcibly, the equipment will consume excessive power and generate excessive heat. Therefore, with state-of-the-art techniques, it is difficult to record video data with a high resolution every frame, and it is possible only to obtain a high-resolution image per a few frames of video data. Researches have been made for the use of various image processing techniques after images are recorded.
More specifically, a conventional technique in the subject field is for obtaining video data of a high resolution in both the time and space domains from video data that has a high resolution in the time domain but a low resolution in the space domain, by using images that have a low resolution in the time domain but a high resolution in the space domain.
Patent Document 1 (Japanese Patent No. 3240339) discloses a technique for producing high-resolution video data based on low-resolution video data and high-resolution still images that are being recorded. With the technique of Patent Document 1, high-resolution still images are associated with samples of frames of low-resolution video data with a predetermined sampling interval therebetween so as to spatially compensate for the low-resolution video data to thereby increase the resolution of the video data.
Patent Document 2 (Japanese National Phase PCT Laid-Open Publication No. 2005-522108) discloses a technique as follows. A scene is recorded to produce low-quality image data while a portion of the scene is recorded to produce high-quality image data. Then, the high-quality image data and the low-quality image data are used as a learning pair in a learning algorithm to thereby determine quality-improving function parameters, based on which the high-quality image of the rest of the scene is derived.
However, the conventional techniques have the following problems.
The technique disclosed in Patent Document 1 is to produce video data of a high resolution in the space domain based on video data of a low resolution. Specifically, video data and still images are associated with each other at discrete points along the time axis, and therefore information for frames with which still images have already been associated is used for video data frames of which there is no association information. Then, similar signal level edges are searched for, which if found are considered to indicate translation-like movement of an object. Then, a motion vector searching process is used to determine the pixels to be compensated for in the space domain. The problem is that the searching process imposes a heavy load, and it may result in incorrect pixel associations. If the object deforms or turns simultaneously with its movement, the process may not find points to be associated with each other, whereby the process fails.
With the technique of Patent Document 1, the high-resolution image and the low-resolution image are read out at the same speed. If the resolution conversion factor between the video data and the still images is up to about two (i.e., two horizontally and two vertically), the process of reading out high-resolution images does not take excessive amounts of time. However, if the conversion factor is increased to about four, the total area of a high-resolution image to be read out is 16 times as great as that of a low-resolution image, and the process of reading out high-resolution images will take excessive amounts of time. As a result, there will be a significant increase in the number of frames to be dropped from the recorded video data, and the quality is likely to deteriorate due to the frame dropping in the video data.
With the technique of Patent Document 2, the position where the high-quality image data is recorded is fixed, for example, substantially at the center of the scene. Therefore, the quality-improving function parameters are determined based on the image characteristics at the fixed position. Thus, high-quality images may not be derived appropriately if the image characteristics at the fixed position are different from those of other positions. This presents a significant problem especially when increasing the resolution of video data, whereby it is likely that a sufficient precision will not be obtained in the resolution conversion.